Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	Jeff Kaplan	Date:	November 15, 2012 06:47:56am
Forum:	web	Subject:	Re: Strange requests from archive.org_bot

what URL are you trying to see?

Reply to this post

Reply [edit]

Poster:	dpavlyuchkov	Date:	November 15, 2012 01:45:41pm
Forum:	web	Subject:	Re: Strange requests from archive.org_bot

I am not trying to see an URL. I just open my access logs from apache and see that archive's bot is browsing the site, to save it, apparently. However the links the bot is trying to visit are really strange - none of them are available at the HTML code after the page renders and none of them make any sense. Just look at them - they are clearly parsed from the javascript with some error or so.

I am wondering what might be wrong because our errors statistics goes batshit crazy because of such requests.

Any insight on this?